Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
Multi-dynamic aware network for unaligned multimodal language sequence sentiment analysis
Junhao LUO, Yan ZHU
Journal of Computer Applications    2024, 44 (1): 79-85.   DOI: 10.11772/j.issn.1001-9081.2023060815
Abstract148)   HTML7)    PDF (1299KB)(99)       Save

Considering the issue that the word alignment methods commonly used in the existing methods for aligned multimodal language sequence sentiment analysis lack interpretability, a Multi-Dynamic Aware Network (MultiDAN) for unaligned multimodal language sequence sentiment analysis was proposed. The core of MultiDAN was multi-layer and multi-angle extraction of dynamics. Firstly, Recurrent Neural Network (RNN) and attention mechanism were used to capture the dynamics within the modalities; secondly, intra- and inter-modal, long- and short-term dynamics were extracted at once using Graph Attention neTwork (GAT); finally, the intra- and inter-modal dynamics of the nodes in the graph were extracted again using a special graph readout method to obtain a unique representation of the multimodal language sequence, and the sentiment score of the sequence was obtained by applying a MultiLayer Perceptron (MLP) classification. The experimental results on two commonly used publicly available datasets, CMU-MOSI and CMU-MOSEI, show that MultiDAN can fully extract the dynamics, and the F1 values of MultiDAN on the two unaligned datasets improve by 0.49 and 0.72 percentage points respectively, compared to the optimal Modal-Temporal Attention Graph (MTAG) in the comparison methods, which have high stability. MultiDAN can improve the performance of sentiment analysis for multimodal language sequences, and the Graph Neural Network (GNN) can effectively extract intra- and inter-modal dynamics.

Table and Figures | Reference | Related Articles | Metrics